<html>
<head><meta charset="utf-8"><title>Allocators with context · t-libs/wg-allocators · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/index.html">t-libs/wg-allocators</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html">Allocators with context</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="166866071"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166866071" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Mena Quintero <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166866071">(May 29 2019 at 23:16)</a>:</h4>
<p>Hi, everyone.  I'm doing an experiment in porting parts of bzip2 (the unmaintained / ubiquitous compression library) to rust.  In there, a bz_stream is a C struct that is the main object one interacts with.  It lets one define a custom allocator by letting the caller set bz_stream-&gt;bzalloc and bz_stream-&gt;bzfree fields.  Internally the library does a few allocations with those, or falls back to malloc()/free() if the caller set those fields to NULL.<br>
I'm looking in <a href="http://codesearch.debian.net" target="_blank" title="http://codesearch.debian.net">codesearch.debian.net</a> to see just what uses a custom allocator with bzip, and it's only low-level stuff like freetype, ImageMagick and python.<br>
Let's say I end up having a libbzip2.a staticlib built from Rust that preserves the C API.  How might I define a Rust allocator that calls into those -&gt;bzalloc and -&gt;bzfree?</p>



<a name="166866162"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166866162" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Mena Quintero <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166866162">(May 29 2019 at 23:18)</a>:</h4>
<p>I think I cannot just set the global allocator, because there's no place to hook into the caller-supplied bz_stream</p>



<a name="166938153"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166938153" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166938153">(May 30 2019 at 19:23)</a>:</h4>
<p>You can "easily" define your own allocator by implementing the <a href="https://doc.rust-lang.org/std/alloc/trait.Alloc.html" target="_blank" title="https://doc.rust-lang.org/std/alloc/trait.Alloc.html"><code>Alloc</code></a> trait. It does require nightly Rust though.</p>



<a name="166938220"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166938220" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166938220">(May 30 2019 at 19:24)</a>:</h4>
<p>The problem comes about with <em>using</em> it, as I don't think any Rust types expose a parameterized allocator.</p>



<a name="166938280"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166938280" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166938280">(May 30 2019 at 19:25)</a>:</h4>
<p><code>RawVec</code> does, but I don't think it's public in any fashion.</p>



<a name="166938367"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166938367" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166938367">(May 30 2019 at 19:26)</a>:</h4>
<p>That being said, I bet you don't actually <em>need</em> to make an allocator (although it would be nice)</p>



<a name="166938393"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166938393" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tim Diekmann <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166938393">(May 30 2019 at 19:27)</a>:</h4>
<p><code>RawVec</code> is feature gated and has an unstable API. I wouldn't use it.</p>



<a name="166938522"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166938522" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166938522">(May 30 2019 at 19:28)</a>:</h4>
<p>A thorough answer would require knowing what kind of allocations you fully need, but you can write your own specific code.</p>



<a name="166938534"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166938534" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tim Diekmann <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166938534">(May 30 2019 at 19:28)</a>:</h4>
<p>Also <code>Alloc</code> will probably be changed as well.</p>



<a name="166938713"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166938713" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166938713">(May 30 2019 at 19:31)</a>:</h4>
<p><span class="user-mention" data-user-id="216785">@Tim Diekmann</span> certainly possible! That being said, <span class="user-mention" data-user-id="223500">@Federico Mena Quintero</span> might be in a good position to help explore some of this API from an angle we might not normally get.</p>



<a name="166938945"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166938945" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166938945">(May 30 2019 at 19:34)</a>:</h4>
<p>It might also be worth exploring if bzip's <code>NULL</code> means "use <code>malloc</code>" or "use <em>some</em> default".</p>



<a name="166939678"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166939678" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166939678">(May 30 2019 at 19:43)</a>:</h4>
<p>You can get reasonably far with appropriate <code>Deref</code> structs though:</p>
<div class="codehilite"><pre><span></span>use libc; // 0.2.51
use std::{
    ffi::c_void,
    ops::{Deref, DerefMut},
    slice,
};

type AllocFn = unsafe extern &quot;C&quot; fn(usize) -&gt; *mut c_void;

#[derive(Debug, Default)]
struct BzipConfig {
    allocator: Option&lt;AllocFn&gt;,
}

impl BzipConfig {
    fn allocator(&amp;self) -&gt; AllocFn {
        self.allocator.unwrap_or(libc::malloc)
    }

    fn allocate_bytes(&amp;self, size: usize) -&gt; DataChunk {
        let alloc = self.allocator();
        unsafe {
            DataChunk {
                data: alloc(size) as *mut u8,
                cap: size,
                size: 0,
            }
        }
    }
}

// TODO: implement Drop!
struct DataChunk {
    data: *mut u8,
    size: usize,
    cap: usize,
}

impl Deref for DataChunk {
    type Target = [u8];

    fn deref(&amp;self) -&gt; &amp;Self::Target {
        unsafe { slice::from_raw_parts(self.data, self.size) }
    }
}

impl DerefMut for DataChunk {
    fn deref_mut(&amp;mut self) -&gt; &amp;mut Self::Target {
        unsafe { slice::from_raw_parts_mut(self.data, self.size) }
    }
}

fn main() {
    let bz = BzipConfig::default();
    let mut d = bz.allocate_bytes(123);
    println!(&quot;{}&quot;, d.len());
}
</pre></div>



<a name="166946397"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166946397" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Mena Quintero <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166946397">(May 30 2019 at 21:08)</a>:</h4>
<p>thanks, I'll see if <code>Alloc</code> does what I need.  bzlib only ever uses the custom allocator for "allocate me space for my private structs", and for a couple of arrays.  I may be able to ignore the caller's wishes and use whatever allocator Rust wants to use :)</p>



<a name="166953183"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/197181-t-libs/wg-allocators/topic/Allocators%20with%20context/near/166953183" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/197181-t-libs/wg-allocators/topic/Allocators.20with.20context.html#166953183">(May 30 2019 at 23:03)</a>:</h4>
<p>I'd certainly go the "ignore" route to start with and get the bulk of the functionality in place.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>